Upward facing light sensor for plant detection

ABSTRACT

A farming machine is configured to identify and treat plants in a field. The farming machine includes one or more light sensors for measuring a characteristic of light. The one or more light sensors are coupled to the farming machine and are directed a substantially upwards orientation away from the plants. A control system adjusts settings of an image acquisition system based on a characteristic of light measured by the one or more light sensors. The image acquisition system captures an image of a plant using one or more image sensors coupled to the farming machine, the one or more image sensors directed in a substantially downwards orientation towards the plants. The control system identifies a plant in the image and actuates a treatment mechanism to treat the identified plant.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/789,406, filed Feb. 12, 2020, now U.S. Pat. No. 11,615,486, which isincorporated by reference in its entirety.

TECHNICAL FIELD

The described subject matter generally relates to identification andtreatment of plants, and more specifically to identification andtreatment of plants in a field using an upward facing light sensor.

BACKGROUND

Conventional systems for treating crops in a field broadly applytreatment to all plants in the field or to entire zones of plants withinthe field. These systems have significant drawbacks as they often applythe same treatment to the entire field of plants. For example, in thecase of a spray type treatment, treatment fluid is applied throughoutthe zone or field, resulting in significant waste. When the treatment isa nitrogen-containing fertilizer, excess treatment is harmful to theenvironment in aggregate. Further, in conventional spray treatmentsystems, crops and weeds are treated collectively. Thus, in the case offertilizer treatments, weeds may benefit from treatment unless separateeffort is expended to remove weeds before treatment.

Currently, it is difficult to apply treatments to individual plantsrather than large areas of the field. In an example, farmers manuallyapply treatment to individual plants which does not affect both weedsand crops. This and similar methods are exceptionally labor-intensiveand costly when performed at industrial scale. While some conventionalfarming systems use imaging technology to identify and treat crops in afield (e.g., satellite imaging, color imaging, thermal imaging, etc.),many of these systems are limited in their ability to properly identifyand treat plants at the individual plant level. For example, satelliteimages have poor resolution for detecting individual plants and colorbased imaging systems treat all green plants equally whether they are aweed or a crop.

SUMMARY

A farming machine identifies a plant for treatment using a lightmeasurement system and an image acquisition system as the farmingmachine travels through a field of plants. A treatment mechanism coupledto the farming machine treats the identified plant as the farmingmachine travels past the identified plant in the field. The lightmeasurement system includes a light sensor coupled to the farmingmachine. The light measurement system includes one or more light sensorsdirected in an upwards orientation away from the plants in the field.The light measurement system measures a characteristic of light (e.g.,intensity, color temperature) incident on the light sensor. The farmingmachine adjusts one or more settings of the image acquisition systembased on the measured characteristic of incident light. The settings mayinclude, for example, ISO, shutter speed, white balance, aperture, etc.The image acquisition system captures an image of a plant using an imagesensor coupled to the farming machine. The image sensor is directed in adownwards orientation towards the plants in the field. A control systemidentifies a plant in the image. The control system actuates thetreatment mechanism to treat the plant identified in the image capturedby the image acquisition system as the farming machine travels past theplant in the field.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1A illustrates a side view of a farming machine, in accordance witha first example embodiment.

FIG. 1B illustrates a front view of a farming machine, in accordancewith the first example embodiment.

FIG. 1C illustrates an isometric view of a farming machine, inaccordance with a second example embodiment.

FIG. 1D illustrates a top view of a farming machine, in accordance withthe second embodiment.

FIG. 1E illustrates an isometric view of a farming machine, inaccordance with a third example embodiment.

FIG. 2 illustrates a system environment for identifying and treating aplant, in accordance with one embodiment.

FIG. 3A illustrates a schematic of a light sensor for receiving andprocessing incident light without a cap, in accordance with oneembodiment.

FIG. 3B illustrates a schematic of a light sensor for receiving andprocessing incident light with a cap, in accordance with one embodiment.

FIG. 4A illustrates a first configuration of a farming machine includinga light sensor and an image sensor, in accordance with one embodiment.

FIG. 4B illustrates a second configuration of a farming machineincluding a light sensor and an image sensor, in accordance with oneembodiment.

FIG. 4C illustrates a third configuration of a farming machine includinga light sensor and an image sensor, in accordance with one embodiment.

FIG. 4D illustrates a fourth configuration of a farming machineincluding a light sensor and an image sensor, in accordance with oneembodiment.

FIG. 5A is a schematic of a first operational condition of a farmingmachine with an array of light sensors, in accordance with oneembodiment.

FIG. 5B is a schematic of a second operational condition of a farmingmachine with an array of light sensors, in accordance with oneembodiment.

FIG. 5C is a schematic of a third operational condition of a farmingmachine with an array of light sensors, in accordance with oneembodiment.

FIG. 6 is a flow chart illustrating a method of identifying and treatinga plant using a farming machine with an upward facing light sensor and adownward facing image sensor, in accordance with one embodiment.

FIG. 7 is a schematic illustrating a control system, in accordance withone embodiment.

The figures and the following description describe certain embodimentsby way of illustration only. One skilled in the art will readilyrecognize from the following description that alternative embodiments ofthe structures and methods illustrated herein may be employed withoutdeparting from the principles described herein. Reference will now bemade to several embodiments, examples of which are illustrated in theaccompanying figures. It is noted that wherever practicable similar orlike reference numbers may be used in the figures and may indicatesimilar or like functionality

DETAILED DESCRIPTION I. Introduction

A farming machine includes an automated or semi-automated system foridentifying and treating plants in a field. The farming machine employsan image acquisition system to identify plants for treatment as thefarming machine travels through the field. A control systemautomatically actuates a treatment mechanism coupled to the farmingmachine to treat an identified plant. As such, the system targets andtreats plants individually, thus reducing waste and preventing weedgrowth resulting from treatments that are applied liberally across afield. Using an automated system also reduces manual labor and othercosts associated with treating plants individually, improving farmingefficiency.

In some example systems, an image acquisition system uses one or moreimage sensors to capture an image of a plant to identify a plant fortreatment. However, the quality of an image captured by an image sensoris often subject to environmental conditions. For example, imagescaptured at different times of day and in different weather conditionsvary depending on the characteristics of the available light. Toillustrate, an image captured on a sunny day appears extremely brightcompared to an image captured on an overcast day. The difference inlight may negatively affect the identification of a plant in the image.For example, a plant in the image may be identified incorrectly (e.g.,identified as a weed when it is a crop), an unhealthy plant may appearhealthy in the image, and a location of the plant in the field may bemisidentified.

Misidentification of plants may lead to a farming machine improperlytreating the misidentified plants. For example, a farming machinecaptures an image of a plant in the early morning such that its colorappears darker in the image than in reality. The farming machineincorrectly identifies the plant as an unhealthy because of itsseemingly dark appearance. The farming machine treats the misidentifiedplant accordingly (e.g., the plant is removed, killed, etc.). However,in appropriate lighting (e.g., later in the day), the farming machinecaptures an image of the plant and its color indicates, appropriately,that it is healthy. In this case, the farming machine does notincorrectly treat the plant. Thus, farming machines includingtraditional image acquisition systems may have limited hours and/orconditions of operation (e.g., from 10 am to 3 pm) such that plants arenot misidentified and mistreated. A farming machine including an imagesystem that determines characteristics of the light at the time offarming allows for improved accuracy of plant identification andtreatment. Furthermore, a farming machine configured to determine lightcharacteristics may lead to increased operational hours for the farmingmachine alongside improved effectiveness and efficiency. Describedherein is a farming machine that uses a light measurement system todetermine settings for an image acquisition system to identify a plantfor treatment.

II. Plant Treatment System

A farming machine that identifies and treats plants may have a varietyof configurations, some of which are described in greater detail below.For example, FIG. 1A is a side view of a first embodiment of a farmingmachine and FIG. 1B is a front view of the first embodiment of thefarming machine of FIG. 1A. FIG. 1C is an isometric view of a secondembodiment of a farming machine and FIG. 1D is a top view of the secondembodiment of the farming machine of FIG. 1C. FIG. 1E is a thirdembodiment of a farming machine, in accordance with one embodiment. Thefarming machine 100, illustrated in FIGS. 1A-1E, includes a detectionmechanism 110, a treatment mechanism 120, and a control system 130. Thefarming machine 100 can additionally include a mounting mechanism 140, averification mechanism 150, a power source, digital memory,communication apparatus, or any other suitable component. The farmingmachine 100 can include additional or fewer components than describedherein. Furthermore, the components of the farming machine 100 can havedifferent or additional functions than described below.

The farming machine 100 functions to apply a treatment to one or moreplants 102 within a geographic area 104. Often, treatments function toregulate plant growth. The treatment is directly applied to a singleplant 102 (e.g., hygroscopic material), but can alternatively bedirectly applied to multiple plants, indirectly applied to one or moreplants, applied to the environment associated with the plant (e.g.,soil, atmosphere, or other suitable portion of the plant environmentadjacent to or connected by an environmental factor, such as wind), orotherwise applied to the plants. Treatments that can be applied includenecrosing the plant, necrosing a portion of the plant (e.g., pruning),regulating plant growth, or any other suitable plant treatment.Necrosing the plant can include dislodging the plant from the supportingsubstrate 106, incinerating a portion of the plant, applying a treatmentconcentration of working fluid (e.g., fertilizer, hormone, water, etc.)to the plant, or treating the plant in any other suitable manner.Regulating plant growth can include promoting plant growth, promotinggrowth of a plant portion, hindering (e.g., retarding) plant or plantportion growth, or otherwise controlling plant growth. Examples ofregulating plant growth includes applying growth hormone to the plant,applying fertilizer to the plant or substrate, applying a diseasetreatment or insect treatment to the plant, electrically stimulating theplant, watering the plant, pruning the plant, or otherwise treating theplant. Plant growth can additionally be regulated by pruning, necrosing,or otherwise treating the plants adjacent the plant.

The plants 102 can be crops, but can alternatively be weeds or any othersuitable plant. The crop may be cotton, but can alternatively belettuce, soy beans, rice, carrots, tomatoes, corn, broccoli, cabbage,potatoes, wheat or any other suitable commercial crop. The plant fieldin which the system is used is an outdoor plant field, but canalternatively be plants within a greenhouse, a laboratory, a grow house,a set of containers, a machine, or any other suitable environment. Theplants are grown in one or more plant rows (e.g., plant beds), whereinthe plant rows are parallel, but can alternatively be grown in a set ofplant pots, wherein the plant pots can be ordered into rows or matricesor be randomly distributed, or be grown in any other suitableconfiguration. The crop rows are generally spaced between 2 inches and45 inches apart (e.g. as determined from the longitudinal row axis), butcan alternatively be spaced any suitable distance apart, or havevariable spacing between multiple rows.

The plants 102 within each plant field, plant row, or plant fieldsubdivision generally includes the same type of crop (e.g. same genus,same species, etc.), but can alternatively include multiple crops (e.g.,a first and a second crop), both of which are to be treated. Each plant102 can include a stem, arranged superior (e.g., above) the substrate106, which supports the branches, leaves, and fruits of the plant. Eachplant can additionally include a root system joined to the stem, locatedinferior the substrate plane (e.g., below ground), that supports theplant position and absorbs nutrients and water from the substrate 106.The plant can be a vascular plant, non-vascular plant, ligneous plant,herbaceous plant, or be any suitable type of plant. The plant can have asingle stem, multiple stems, or any number of stems. The plant can havea tap root system or a fibrous root system. The substrate 106 is soil,but can alternatively be a sponge or any other suitable substrate.

The detection mechanism 110 is configured to identify a plant fortreatment. As such, the detection mechanism 110 can include one or moresensors for identifying a plant. For example, the detection mechanism110 can include a multispectral camera, a stereo camera, a CCD camera, asingle lens camera, hyperspectral imaging system, LIDAR system (lightdetection and ranging system), dyanmometer, IR camera, thermal camera,humidity sensor, light sensor, temperature sensor, or any other suitablesensor. In the embodiment of FIGS. 2-5 and described in greater detailbelow, the detection mechanism 110 includes an image sensor configuredto capture an image of a plant. In some example systems, the detectionmechanism 110 is mounted to the mounting mechanism 140, such that thedetection mechanism 110 traverses over a geographic location before thetreatment mechanism 120 as the farming machine 100 moves traversesthrough the geographic location. However, in some embodiments, thedetection mechanism 110 traverses over a geographic location atsubstantially the same time as the treatment mechanism 120. In anembodiment of the farming machine 100, the detection mechanism 110 isstatically mounted to the mounting mechanism 140 proximal the treatmentmechanism 120 relative to the direction of travel 115. In other systems,the detection mechanism 110 can be incorporated into any other componentof the farming machine 100.

The treatment mechanism 120 functions to apply a treatment to anidentified plant 102. The treatment mechanism 120 applies the treatmentto the treatment area 122 as the farming machine 100 moves in adirection of travel 115. The effect of the treatment can include plantnecrosis, plant growth stimulation, plant portion necrosis or removal,plant portion growth stimulation, or any other suitable treatment effectas described above. The treatment can include plant 102 dislodgementfrom the substrate 106, severing the plant (e.g., cutting), plantincineration, electrical stimulation of the plant, fertilizer or growthhormone application to the plant, watering the plant, light or otherradiation application to the plant, injecting one or more working fluidsinto the substrate 106 adjacent the plant (e.g., within a thresholddistance from the plant), or otherwise treating the plant. The treatmentmechanism 120 is operable between a standby mode, wherein the treatmentmechanism 120 does not apply a treatment, and a treatment mode, whereinthe treatment mechanism 120 is controlled by the control system 130 toapply the treatment. However, the treatment mechanism 120 can beoperable in any other suitable number of operation modes.

The farming machine 100 may include one or more treatment mechanisms120. A treatment mechanism 120 may be fixed (e.g., statically coupled)to the mounting mechanism 140 or attached to the farming machine 100relative to the detection mechanism 110. Alternatively, the treatmentmechanism 120 can rotate or translate relative to the detectionmechanism 110 and/or mounting mechanism 140. In one variation, such asin FIGS. 1A-1B, the farming machine 100 a includes a single treatmentmechanism, wherein the treatment mechanism 120 is actuated or thefarming machine 100 a moved to align the treatment mechanism 120 activearea 122 with the targeted plant 102. In a second variation, the farmingmachine 100 includes an assembly of treatment mechanisms, wherein atreatment mechanism 120 (or subcomponent of the treatment mechanism 120)of the assembly is selected to apply the treatment to the identifiedplant 102 or portion of a plant in response to identification of theplant and the plant position relative to the assembly. In a thirdvariation shown, such as in FIGS. 1C-1E, the farming machine (i.e., 100b, 100 c) includes an array of treatment mechanisms 120, wherein thetreatment mechanisms 120 are actuated or the farming machine (i.e., 100b, 100 c) is moved to align the treatment mechanism 120 active areas 122with the targeted plant 102 or plant segment.

The farming machine 100 includes a control system 130 for controllingoperations of system components. The control system can receiveinformation from and/or provide input to the detection mechanism 110,the verification mechanism 150, and the treatment mechanism 120. In someembodiments, the control system 130 may be configured to controloperating parameters of the farming machine 100 (e.g., speed,direction). The control system 130 also controls operating parameters ofthe detection mechanism 110. Operating parameters of the detectionmechanism 110 may include processing time, location and/or angle of thedetection mechanism 110, image capture intervals, image capturesettings, etc. The control system 130 may be a computer, as described ingreater detail below in relation to FIG. 7 . The control system 130 maybe coupled to the farming machine 100 such that an operator (e.g., adriver) can interact with the control system 130. In other embodiments,the control system 130 is physically removed from the farming machine100 and communicates with system components (e.g., detection mechanism110, treatment mechanism 120, etc.) wirelessly.

In some configurations, the farming machine 100 includes a mountingmechanism 140 that functions to provide a mounting point for the systemcomponents. In one example, as shown in FIG. 1A-1B, the mountingmechanism 140 statically retains and mechanically supports the positionsof the detection mechanism 110, the treatment mechanism 120, and theverification mechanism 150 relative to a longitudinal axis of themounting mechanism 140. The mounting mechanism 140 is a chassis orframe, but can alternatively be any other suitable mounting mechanism.In the embodiment of FIGS. 1C-1E, the mounting mechanism 140 extendsoutward from a body of the farming machine (i.e., 100 b, 100 c) in thepositive and negative x-direction (in the illustrated orientation ofFIGS. 1A-1E) such that the mounting mechanism 140 is approximatelyperpendicular to the direction of travel 115. The mounting mechanism 140in FIGS. 1C-1E includes an array of treatment mechanisms 120 positionedlaterally along the mounting mechanism 140. In alternate configurations,there may be no mounting mechanism 140, the mounting mechanism 140 maybe alternatively positioned, or the mounting mechanism 140 may beincorporated into any other component of the farming machine 100.

The farming machine 100 includes a first set of coaxial wheels and asecond set of coaxial wheels, wherein the rotational axis of the secondset of wheels is parallel with the rotational axis of the first set ofwheels. In the first embodiment, each wheel in each set is arrangedalong an opposing side of the mounting mechanism 140 such that therotational axes of the wheels are approximately perpendicular to themounting mechanism 140. In the second and third embodiments of thefarming machine, the rotational axes of the wheels are approximatelyparallel to the mounting mechanism 140. In alternative embodiments, thesystem can include any suitable number of wheels in any suitableconfiguration. The farming machine 100 may also include a couplingmechanism 142, such as a hitch, that functions to removably orstatically couple to a drive mechanism, such as a tractor, more to therear of the drive mechanism (such that the farming machine 100 isdragged behind the drive mechanism), but can alternatively be attachedto the front of the drive mechanism or to the side of the drivemechanism. Alternatively, the farming machine 100 can include the drivemechanism (e.g., a motor and drive train coupled to the first and/orsecond set of wheels). In other example systems, the system may have anyother means of traversing through the field.

In some configurations, the farming machine 100 additionally include averification mechanism 150 that functions to record a measurement of theambient environment of the farming machine 100. The farming machine maybe use the measurement to verify or determine the extent of planttreatment. The verification mechanism 150 records a measurement of thegeographic area previously measured by the detection mechanism 110. Theverification mechanism 150 records a measurement of the geographicregion encompassing the plant treated by the treatment mechanism 120.The verification mechanism 150 measurement can additionally be used toempirically determine (e.g., calibrate) treatment mechanism operationparameters to obtain the desired treatment effect. The verificationmechanism 150 can be substantially similar (e.g., be the same type ofmechanism as) the detection mechanism 110, or can be different from thedetection mechanism 110. In some embodiments, the verification mechanism150 is arranged distal the detection mechanism 110 relative thedirection of travel, with the treatment mechanism 120 arranged therebetween, such that the verification mechanism 150 traverses over thegeographic location after treatment mechanism 120 traversal. However,the mounting mechanism 140 can retain the relative positions of thesystem components in any other suitable configuration. In otherconfigurations of the farming machine 100, the verification mechanism150 can be included in other components of the system.

In some configurations, the farming machine 100 may additionally includea power source, which functions to power the system components,including the detection mechanism 110, control system 130, and treatmentmechanism 120. The power source can be mounted to the mounting mechanism140, can be removably coupled to the mounting mechanism 140, or can beseparate from the system (e.g., located on the drive mechanism). Thepower source can be a rechargeable power source (e.g., a set ofrechargeable batteries), an energy harvesting power source (e.g., asolar system), a fuel consuming power source (e.g., a set of fuel cellsor an internal combustion system), or any other suitable power source.In other configurations, the power source can be incorporated into anyother component of the farming machine 100.

In some configurations, the farming machine 100 may additionally includea communication apparatus, which functions to communicate (e.g., sendand/or receive) data between the control system 130 and a set of remotedevices. The communication apparatus can be a Wi-Fi communicationsystem, a cellular communication system, a short-range communicationsystem (e.g., Bluetooth, NFC, etc.), or any other suitable communicationsystem.

III. System Environment

As described above, a farming machine (e.g., farming machine 100) mayoperate in different conditions (e.g., at different times of day, indifferent weather conditions, etc.) and the characteristics of light atthe time of operation may vary. The characteristics of light may affectoperation of the detection mechanism 110, which, in turn, affects how afarming machine identifies and treats plants. As such, a farming machinemay include a light measurement system configured to evaluatecharacteristics of light in order to improve plant identification andtreatment.

FIG. 2 illustrates a system environment for identifying and treatingplants, according to one embodiment. The system environment 200 includesa light measurement system 210, an image acquisition system 220, acontrol system 130, and a farming machine 240 (e.g., farming machine100) connected via a Controller Area Network (CAN) bus 250. Inalternative embodiments, the system environment 200 includes additionalor fewer components than described herein. The functions of thecomponents may also be distributed in a different manner than describedbelow.

The light measurement system 210 is configured to receive and processlight incident on a light sensor (“incident light”). The lightmeasurement system 210 includes a light reception module 212 and a lightprocessing module 214. The light reception module 212 includes a lightsensor and detects incident light. The light reception module 212 isdescribed in greater detail below in relation to FIG. 3 . In someembodiments, the light reception module 212 detects incident light in avisible and/or infrared spectrum, but other spectrums of electromagneticradiation are also possible (e.g., ultraviolet). The light processingmodule 214 analyzes one or more characteristics of the incident light.The characteristics can include, for example, intensity, brightness, andcolor temperature. In alternative embodiments, the light processingmodule 214 evaluates fewer or additional characteristics of incidentlight. The light measurement system 210 provides one or more measuredcharacteristics of incident light to the image acquisition system 220and/or the control system 130 via the CAN bus 250.

The image acquisition system 220 is configured to capture an image of aplant. The image acquisition system 220 includes a setting managementmodule 222, an image capture module 224, and an image processing module226. In alternative embodiments, the image acquisition system 220includes fewer or greater components than described herein.

The setting management module 222 is configured to adjust one or moresettings of the image capture module 224. Settings are selectableparameters that, when changed, alter a configuration of the image sensorand, thereby, change how an image sensor captures an image. Generally,settings are selected such that the image sensor captures an image witha proper exposure. Exposure is the amount of light per unit area on animage sensor and may affect how light or dark an image appears. Here, aproper exposure is an exposure that allows the farming machine toaccurately identify a plant in the image.

In some examples, the setting management module 222 adjusts exposuresettings (e.g., aperture, shutter speed, ISO speed) and white balance.In other examples, the setting management module 222 is configured toadjust additional settings (e.g., focus, flash, zoom, etc.) of the imagecapture module 224. The setting management module 222 may be configuredto adjust one or more settings at regular time intervals, at a specifiedtime, each time the system resets, randomly, based on input from thecontrol system 130, or according to other suitable methods.

In one embodiment, the setting management module 222 adjusts one or moresettings based on a measured characteristic of light determined by thelight measurement system 210. For example, the light measurement system210 measures an intensity of 120,000 lux (e.g., on a sunny day), and inresponse, the setting management module 222 reduces the shutter speed ofthe image capture module 224. In another example, the light measurementsystem 210 measures a cool color temperature (e.g., on an overcast day),and the setting management module 222 adjusts the white balance of theimage capture module 224 to a warmer temperature. In some embodiments,the setting management module 222 adjusts the settings of the imagecapture module 224 responsive to detecting a change in a characteristicof light measured by the light measurement system 210. The settingmanagement module 222 may also adjust settings of the image capturemodule 224 based on additional criteria. Some additional examples andcriteria are described in greater detail below.

In some embodiments, the setting management module 222 adjusts thesettings of the image capture module 224 according to specifiedsettings. Specified settings are one or more settings for an imagesensor that the setting management module 222 implements in specificcircumstances. For example, the setting management module 222 implementsa specific setting when the light measurement system 210 detects aspecific measured characteristic of light. To illustrate, the imageacquisition system 220 includes (i.e., in a datastore) a set ofintensity ranges, with each intensity range including a range of lightintensities. Each intensity range corresponds to a specific setting foran image sensor. The light measurement system 210 measures an intensityof light and the setting management module 222 compares the measuredintensity to the set of intensity ranges. The setting management module222 adjusts the settings of the light measurement system based on thespecified setting associated with the intensity range including themeasured intensity. To provide additional context, the light measurementsystem 210 measures an intensity of 95,000 lux, and the settingmanagement module 222 compares the measured intensity to a set ofintensity ranges. The setting management module 222 determines that themeasured intensity in the range of 90,000 to 110,000 lux, whichcorresponds a shutter speed of 0.05 seconds. Thus, the settingmanagement module 222 adjusts the shutter speed of the image capturemodule 224 to 0.05 seconds.

In other examples, the setting management module 222 compares a measuredcharacteristic of light to a typical value for the characteristic andadjusts a setting according to one or more adjustment thresholds. Thetypical value of a characteristic can be an average value for anoperating condition (e.g., on a sunny day, on a cloudy day, at a certaintime of day, etc.), a randomly assigned value, a default value, a valuedetermined by the control system 130, or any other appropriate value.Responsive to a difference between the measured characteristic and acorresponding typical value exceeding an adjustment threshold, thesetting management module 222 adjusts one or more settings. For example,a typical value of intensity is 50,000 lux, and the adjustment thresholdis 20,000 lux. The light measurement system 210 measures an incidentintensity of 90,000 lux. The setting management module 222 determinesthat the difference between the measured value and the typical valueexceeds the adjustment threshold and adjusts a setting (e.g., aperture,ISO speed, etc.) accordingly. On the other hand, responsive todetermining the difference does not meet the adjustment threshold, thesetting management module 222 ignores the difference (i.e., the settingsare not changed). In other embodiments, the setting management module222 can adjust one or more settings according to other suitableparameters.

The image capture module 224 is configured to capture an image of aplant for treatment. The image capture module 224 has initial settings(e.g., default settings when the system turns on, resets, etc.) that canbe adjusted by the setting management module 222. In some embodiments,initial settings of the image capture module 224 are settingscorresponding to an operational condition (e.g., settings for a typicalsunny day in the spring, for a certain time of day, etc.). In somecases, an operator of the farming machine may input the operationalcondition such that the initial settings are appropriate. In otherembodiments, the initial settings can be randomly assigned, set by thecontrol system 130, set by an operator, or determined according to anyappropriate method. The setting management module 222 continuouslymonitors and adjusts the settings of the image capture module 224 duringoperation. The image capture module 224 can be configured to capture animage(s) at specified intervals, at a specified time, at random timeintervals, as determined by the control system 130, or according to anyother suitable guidelines.

The image processing module 226 processes an image captured by the imagecapture module 224. Processing the image includes evaluating and/ormodifying image qualities, identifying a plant in the image, and/orevaluating plant properties. Modifying the image can include resizing,debayering, cropping, value normalization, and adjusting image qualitiessuch as contrast, brightness, exposure, temperature, etc. Identifyingthe plant can include determining the type of plant and/or the locationof the plant. In some embodiments, evaluating plant properties alsoincludes determining other characteristics of the plant identified inthe image and adjusting settings appropriately. Some characteristics ofthe plant may include, for example, Normalized Difference VegetationIndex (NDVI), Transformed Chlorophyll Absorption in Reflectance Indexnormalized by Optimized Soil-Adjusted Vegetation Index (TCARI/OSAVI),Normalized Difference Red Edge Index (NDRE), Canopy Chlorophyll ContentIndex (CCCI), Photochemical Reflectance Index (PRI), etc. Additionally,the image processing module 226 can evaluate plant properties todetermine if the plant is healthy and/or if the plant needs treatment.In an embodiment, the image processing module employs a plantidentification model to identify a plant, a plant type, plant features,etc. An example of a plant identification model employed by the imageprocessing model is described in U.S. patent application Ser. No.16/126,842 titled “Semantic Segmentation to Identify and Treat Plants ina Field and Verify the Plant Treatments,” filed on Sep. 10, 2018, butother plant identification models are also possible.

In an embodiment, evaluating plant properties includes determining aperceived reflectance of an identified plant and comparing the perceivedreflectance with an actual reflectance of the identified plant. Actualreflectance is a constant material property (e.g., a plant has a knownreflectance value). However, a perceived reflectance of a plant variesbased on the available light. The image processing module 226 mayincorrectly identify a plant and/or plant characteristics if theperceived reflectance is different from the actual reflectance. Forexample, on a sunny day a plant may have a perceived reflectance greaterthan its actual reflectance and the image processing modulemisidentifies the plant's type. Accordingly, the image processing module226 is configured to determine a difference between the actualreflectance and the perceived reflectance of an identified plant(“reflectance difference”). Determining a reflectance difference allowsthe image processing module 226 to improve plant identification andtreatment in different operating conditions. In an embodiment, the imageprocessing module 226 determines a reflectance difference by comparingthe reflectance difference to a difference threshold. A differencethreshold is a quantification of reflectance difference that causes theimage processing module 226 to incorrectly identify plants. Responsiveto the reflectance difference exceeding the difference threshold, theimage processing module 226 recommends adjusting one or more settings ofthe image acquisition system 220. The setting management module 222adjusts the settings of the image acquisition system 220 based on thereflectance difference of a target plant. Responsive to determining thereflectance difference does not meet the difference threshold, the imageprocessing module 226 does not adjust the settings of the imageacquisition system and/or transmits the reflectance difference to thecontrol system 130 for further evaluation.

The control system 130, described above in relation to FIGS. 1A-1E,communicates with the light measurement system 210, the imageacquisition system 220, and the farming machine 240 to control treatmentof an identified plant. Specifically, the control system 130 evaluates acharacteristic of light using the light measurement system 210 andcaptures an image using the image acquisition system 220. The controlsystem 130 also communicates with the farming machine 240 via the CANbus 250 to treat an identified plant. In some embodiments, the controlsystem 130 actuates one or more treatment mechanisms 244. Instructionsgenerated by the control system 130 may be transmitted to a treatmentmechanism 244 using ethernet connections, CAN bus 250 connections, oranother transmission protocol. In various embodiments, the controlsystem 130 may perform actions prescribed to other systems and modulesin the system environment 200. For example, the control system 130 mayidentify a plant rather than the image processing module 226. Similarly,the control system 130 may determine a setting for an image acquisitionsystem 220. The control system 130 is described in greater detail belowin relation to FIG. 7 .

The farming machine 240 includes an operation module 242 and a treatmentmechanism 244. The farming machine 240 may be any of the farmingmachines 100 described in relation to FIG. 1A-1E, or some other farmingmachine. The operation module 242 controls the operations (e.g., speed,direction) of the farming machine. In some embodiments, a user (e.g., adriver of a farming machine) provides input to the operation module 242to affect operation of the farming machine 240, the light measurementsystem 210, or the image acquisition system 220. In other embodiments,the control system 130 controls the operation module 242 such that thefarming machine 100 is semi-autonomous or autonomous (e.g., operateswithout user input). Additionally, the control system 130 controls thetreatment mechanism 244 to treat identified plants. The treatmentmechanism 244 can be a plurality of treatment devices and types, asdescribed above in relation to FIGS. 1A-1E.

The Controller Area Network (CAN) bus 250 connects nodes of the systemenvironment 200 to allow microcontrollers and devices to communicatewith each other. In some embodiments, the components connected to theCAN bus 250 each have an input and output connection, and the CAN bus250 acts as a translation mechanism for the components. For example, theCAN bus 250 receives input information from the light measurement system210, processes the information, and transmits the information to thecontrol system 130. The control system 130 determines a settingadjustment for the image acquisition system 220 based on the informationreceived from the light measurement system 210 and transmits the settingadjustment to the image acquisition system 220. Further, the CAN bus 250receives image data from the image acquisition system 220 and transmitsimage data to the control system 130. Based on the received information,the control system 130 selects one or more treatment mechanisms 244 tobe actuated. The CAN bus 250 receives and transmits the information tothe farming machine 240 in order to actuate a treatment mechanism 244.The CAN bus 250 can be any suitable network, such as the Internet, aLAN, a MAN, a WAN, a mobile wired or wireless network, a privatenetwork, a virtual private network, a direct communication line, and thelike. The CAN bus 250 can also be a combination of multiple differentnetworks of the same or different types.

IV. System Hardware

As mentioned above, the light measurement system 210 includes at leastone light sensor to receive light. In some embodiments, the lightmeasurement system 210 includes additional light detectors, lightmeasurement optics, fibers, etc. for measuring and processing incidentlight. FIGS. 3A-3B illustrate a schematic of a light sensor 300,according to an embodiment. The light sensor 300 includes a sensingdevice 360, an electronic subsystem 362, a base 364, a wire 366, and acap 368. In alternative embodiments, the light sensor 300 includes feweror greater elements than described below.

The light sensor 300 includes a sensing device 360 for converting lightincident upon the sensing device (“incident light 370” or simply “light370”) into signals representative of the light. The sensing device mayinclude one or more pixels/detectors depending on the configuration. Thesensing device 360 evaluates various spectrums of light. In an example,the sensing device evaluates visible, infrared, and/or ultravioletlight. The sensing device 360 measures light intensity and converts themeasured light intensity to a digital output signal. The sensing device360 can also include a color sensing feature to measure colortemperature. In the embodiment of FIGS. 3A-3B, the sensing device is arectangular prism with a broad surface 361 configured to receive light370. The broad surface 361 is upward facing, described in greater detailbelow, to receive available light 370. The sensing device 360 isapproximately 1.5-2.5 millimeters wide, 1.5-2.5 millimeters long, and0.40-0.80 millimeters tall. In alternative embodiments, the sensingdevice 360 can have any suitable shape and dimensions for receivingincident light 370.

The sensing device 360 is coupled (e.g., electronically, mechanically)to the electronic subsystem 362. The electronic subsystem 362 allows forsignal and power transmission between devices. In the embodiment ofFIGS. 3A-3B, an inferior surface of the sensing device 360 (e.g., asurface opposing the broad surface 361) is coupled to the electronicsubsystem 362. In some embodiments, the electronic subsystem 362 is aprinted circuit board (PCB). In alternative embodiments, the electronicsubsystem 362 can be any element suitable for supporting the function ofthe sensing device 360.

The electronic subsystem 362 is coupled to the CAN bus 250 via a wire366. The wire 366 functions to transmit signals between components. Inthe embodiment of FIGS. 3A-3B, the wire 366 is an electricallyconductive wire encased by an insulative material for protection fromexternal components and operating conditions (e.g., rain). In otherembodiments, the wire 366 is composed of a different material for signaltransmission and/or the light sensor 300 includes additional wires forconnecting the sensing device 360 and/or the electronic subsystem 362 toadditional components (e.g., control system 130, image acquisitionsystem 220, a power source, a computer, etc.). In one example, thesensing device 360 is coupled to an optic fiber for transmitting lightsignals to the control system 130 for processing. In alternativeembodiments, the electronic subsystem 362 is in wireless communicationwith the CAN bus 250 or is connected to a different component via thewire 366.

The electronic subsystem 362 and the sensing device 360 are coupled to acylindrical base 364 for support. In alternative embodiments, the base364 can be a rectangular prism, a trapezoidal prism, a parallelepiped,or any other shape suitable for supporting the electronic subsystem 362and the sensing device 360. The base 364 can include a receptacle forretaining the electronic subsystem 362 and the sensing device 360 inposition. As such, the sensing device 360 and the electronic subsystem362 can be recessed below a surface of the base 364 in the receptacle.In alternative embodiments, the electronic subsystem 362 and/or thesensing device 360 are protruding from the base 364. In the embodimentof FIGS. 3A-3B, the base 364 has a height of approximately 0.6 to 2millimeters and a diameter of 2 to 4 millimeters. Alternatively, thebase 364 can have any suitable dimensions for supporting the sensingdevice 360 and/or the electronic subsystem 362.

In FIGS. 3A-3B, the base 364 is composed of an insulative material(e.g., rubber, plastic, etc.) to prevent undesired signal transmissionbetween the sensing device 360, the electronic subsystem 362, thefarming machine 100, and other system components. The base 364 is alsocomposed of a water resistant and elastic material for protectingelectronic components from damage. Alternatively, the base 364 can becomposed of any material suitable for supporting the other components ofthe light sensor 300. The base 364 is attached to a farming machine 100by a coupling surface 369 (e.g., a bottom surface) such that the broadsurface 361 of the sensing device 360 is upward facing. The couplingsurface 369 can be permanently coupled to the farming machine 100 (e.g.,welded, glued, etc.) or the coupling surface 369 can be temporarilyattached such that the light sensor 300 can be removed for repair,replacement, etc. In other embodiments, the coupling surface 369 can bea different surface of the base 364.

As shown in FIG. 3B, a cap 368 may be coupled to the base 364 forshielding the electronic subsystem 362 and the sensing device 360 fromthe environment. In the embodiment of FIG. 3B, the cap 368 has ahemispheric shape and is composed of a translucent material (e.g.,glass, plastic, etc.). The cap 368 is also composed of a material thatis water resistant and elastic for protecting the sensing device 360 andthe electronic subsystem 362 from the environment. The cap 368 may beremovably coupled to the sensing device 360 such that elements of thelight sensor 300 can be easily replaced or repaired. In otherembodiments, the cap 368 can have any other shape and can be composed ofany material suitable for protecting the sensing device 360 and theelectronic subsystem 362, and/or may be permanently coupled to the base364. In one example, the cap 368 is flush with a surface of the base 364for shielding and/or retaining the sensing device 360 in position (e.g.,within a receptacle of the base 364). Furthermore, in some embodiments,the cap 368 can encapsulate the sensing device 360, the electronicsubsystem 362 and the base 364. Additionally, in some embodiments, thelight sensor 300 may not include a cap 368.

Returning briefly to FIG. 2 , the light measurement system 210 includesone or more light sensors 300 described in FIGS. 3A and 3B. In someembodiments, the light measurement system 210 includes an array of lightsensors. Similarly, the image acquisition system 220 includes one ormore image sensors. In general, an image sensor detects and conveysinformation used to make an image. An image sensor can be a digitalcamera, a camera module, a thermal imaging device, a radar, etc. In someembodiments, the image acquisition system 220 includes an array ofcameras. The cameras can have various capabilities and may be configuredto capture various light spectra. Examples include, but are not limitedto, RGB cameras, near infrared (e.g., red edge or short wave infrared)cameras, ultraviolet cameras, and multi-spectral cameras. The camerasgenerally use CMOS digital image sensors, but may also be CCD imagesensors. The image acquisition system 220 can also include alternatereceptors, focusing optics, data connections, etc. for capturing animage suitable for identifying a plant.

V. System Embodiments

A light sensor 300 included in a light measurement system 210 and animage sensor included in an image acquisition system 220 can beconfigured to a variety of systems. In one embodiment, components of thelight measurement system 210 and components of the image acquisitionsystem 220 are coupled to a farming machine (e.g., farming machine 100)to identify plants as the farming machine moves through a field. FIGS.4A-4D illustrate various configurations of a farming machine (e.g., 400a, 400 b, 400 c, 400 d) including a light measurement system 210 with atleast one light sensor (e.g., 410 a, 410 b, 410 c, and 410 d) and animage acquisition system 220 with at least one image sensor (e.g., 420a, 420 b, 420 c, and 420 d), in accordance with several embodiments.

In the embodiments of FIGS. 4A-4D, the farming machines aresubstantially similar to each other and to farming machine 100, however,each farming machine illustrates a different configuration of lightsensors and image sensors, described in greater detail below. Eachfarming machine (e.g., 400 a, 400 b, 400 c, 400 d) includes a mountingmechanism 140 illustrated as two segments extending laterally from thebody of the farming machine. The farming machine moves in a positivez-direction (e.g., towards the reader) such that the mounting mechanismis approximately perpendicular to the direction of travel. The farmingmachine (e.g., 400 a, 400 b, etc.) includes a control system 130 forcontrolling the image acquisition system 220 and the light measurementsystem 210.

In the illustrated examples, the light sensor(s) are positioned tomeasure incident light and the image sensor(s) are positioned to captureimages of plants in the field. The farming machine changes the settingsof the image sensor(s) based on incident light measured by the lightsensors to increase the quality of images captured by the imagesensor(s). Higher quality images allow the farming machine to moreaccurately identify and treat plants in the field.

In the illustrated examples, a light sensor 410 is illustrated as a boxwith a solid white fill. Each light sensor is coupled to a surface ofthe farming machine such that the light sensor is directed in an upwardsorientation away from the plants in the field. Here, upwards isrepresented by a value of a relative angle between a vector normal to asurface (e.g., broad surface 361) of a sensing device (e.g., sensingdevice 360) in the light sensor (e.g., light sensor 300) and thepositive y-axis (e.g., upwards in the plane of the page for theillustrated examples). In an example, upwards may be any value of therelative angle less than 90°. In other examples, upwards may be a valueof the relative angle less than a threshold angle (e.g., 30°, 40°, 50°,60°, etc.). Whatever the orientation, the light sensor is positioned tomeasure incident light that, when processed by the farming machine,indicates one or more settings of the image sensor to change in order tocapture higher quality images of plants in the field. In someembodiments, the control system 130 adjusts the angle of a light sensor410 based on operating conditions. For example, the control system 130sets the angle to 80° in the morning (i.e., when the sun is low in thesky) and to an angle of 10° during mid-day (i.e., when the sun is highin the sky).

In the illustrated examples, an image sensor 420 is illustrated as a boxwith a cross-hatch pattern. Each image sensor is coupled to a surface ofthe farming machine such that the image sensor is directed in adownwards orientation towards the plants in the field. Downwards isrepresented by a value of a relative angle between a vector normal to alens of an image sensor and the negative y-axis. In one example,downwards may be any value of the relative angle less than 90°. In otherexamples, downwards may be a value of the relative angle less than athreshold angle (e.g., 30°, 40°, 50°, 60°, etc.). The image sensor isoriented such that it captures an image of one or more plants in thefield as the farming machine travels through a field. In someembodiments, the control system 130 adjusts the angle of an image sensor420 based on an operating condition. For example, the control system 130sets the angle to 10° in bright conditions and an angle of 25° in cloudyconditions. The farming machine may identify plants in the image, andtreat the identified plants accordingly.

FIG. 4A illustrates a first embodiment of a farming machine including anupward facing light sensor and downward facing image sensors. Thefarming machine 400 a includes eight image sensors 420 and one lightsensor 410. The image sensors 420 are positioned such that four imagesensors are located laterally along a front surface the mountingmechanism 140. The location of the image sensors 420 on the mountingmechanism 140 is approximately symmetric along a midline of the farmingmachine 400 a. The light sensor 410 is coupled to a superior surface(e.g., top) of the farming machine 100 and is approximately aligned withthe midline of the farming machine 400. In the embodiment of FIG. 4A,the control system 130 adjusts the settings of the array of imagesensors 420 a based on the light measured by the light sensor 410. Assuch, the image sensors 420 operate in a collective manner. Here,operating in a collective manner indicates that the control system 130makes the same setting adjustment for each image sensor 420 coupled tothe farming machine 400 based on the light measurement determined by thelight sensor 410.

FIG. 4B illustrates a second embodiment of the farming machine includingupward facing light sensors and downward facing image sensors. Thefarming machine 400 b includes two light sensors 410 and two imagesensors 420. A light sensor 410 is coupled to a superior surface of themounting mechanism 140 at a mid-region of the mounting mechanism 140. Animage sensor 420 is coupled to a superior surface of the mountingmechanism 140 adjacent to a light sensor 410 of the mounting mechanism140. In the embodiment of FIG. 4B, each light sensor 410 corresponds toan image sensor 420 (a “corresponding pair”). FIG. 4B illustrates twocorresponding pairs (e.g., 430 a, 430 b) of light sensors 410 and imagesensors 420. As a particular example, the corresponding pair 430 aincludes a particular image sensor 420 a and a particular light sensor410 a. Settings of an image sensor 420 are adjusted based on the lightmeasured by the light sensor 410 in its corresponding pair 430. Forexample, the control system 130 adjusts the settings of the image sensor420 a in the corresponding pair 430 a based on incident light measuredby the light sensor 410 a in the corresponding pair 430 a.

FIG. 4C illustrates a third embodiment of the farming machine includingupward facing light sensors and downward facing image sensors. Thefarming machine 400 c includes three light sensors and two imagesensors. A first and second light sensor are coupled to each free end ofthe mounting mechanism 140, respectively. A third light sensor iscoupled to a superior surface of the farming machine 400 c atapproximately a midline of the farming machine 400 c. The image sensors420 are located laterally along an inferior surface of the mountingmechanism 140. In the illustrated example, there is a single lightsensor 410 and a single image sensor 420 on each segment of the mountingmechanism 140. In an embodiment, the light sensor 410 and image sensor420 on each segment of the mounting mechanism 140 may form acorresponding pair similar to above. However, in an embodiment, thelight sensors 410 and image sensors 420 of the farming machine 400 c actin conjunction. That is, the farming machine 400 c uses measuredcharacteristics from all of the light sensors 410 to control the imagesensors 420. For example, a control system 130 evaluates acharacteristic of light (e.g., intensity) measured by each of the threelight sensors 410 and determines an average of the characteristic oflight based on the three light sensors 410. The control system 130adjusts one or more settings of the image sensors 420 based on theaverage. In alternative embodiments, the control system 130 selects oneor more of the light sensors 410 to determine a characteristic of light,described in greater detail below in relation to FIG. 5C.

FIG. 4D illustrates a fourth embodiment of the farming machine includingan upward facing light sensor and downward facing image sensors. Thefarming machine 400 d includes one light sensor 410 and nine imagesensors 420. Eight image sensors 420 are attached laterally along themounting mechanism 140. An image sensor 420 and the light sensor 410 arecoupled to a rod 415 extending from the farming machine 400 d. The rod415 extends approximately orthogonal to the mounting mechanism 140 toseparate the light sensor 410 from other components and/or theenvironment (e.g. to keep the light sensor 410 d away from dirt, frominterfering with a treatment mechanism, etc.). The rod 415 isapproximately aligned with a midline of the farming machine 400 d. Inone example, the rod 415 is a metal rod such that the rod issubstantially straight during different operational conditions. In otherexamples, the rod 415 may be flexible such that the light sensor 410 dcan be angled with respect to the sun (e.g., a broad surface 361 of thelight sensor is angled with respect to an azimuth angle of the sun, orby an angle selected by an operator of the farming machine). In otherembodiments, the rod can be placed in any other location along thefarming machine 400 d (e.g., extending from the mounting mechanism 140,at the rear of the farming machine 400 d, etc.). Additionally, one ormore of the image sensors 420 may also be attached to one or moreadditional rods such that they extend away from the mounting mechanism140.

FIGS. 4A-4D illustrate a variety of configurations of light sensors 410and image sensors 420. In alternative embodiments, other configurationsof light sensors and image sensors may be used for plant identification.For example, the configuration of light sensors shown in FIG. 4C may becombined with the configuration of image sensors shown in FIG. 4D.Furthermore, in some embodiments, light sensors and/or image sensors maybe coupled to a different machine. For example, a light sensor may beattached to a drone or located in a stationary position (e.g., on apost, on a greenhouse, on a shed) and connected wirelessly to thecontrol system 130. In other embodiments, an image sensor and a lightsensor can be coupled to any surface (e.g., front, back, side, top) andany component of a farming machine provided each sensor is facing theappropriate direction (e.g., a light sensor is facing away from theplants and an image sensor is facing towards the plants). Configurationsof light sensors and image sensors may be optimized for effectivelyidentifying and treating plants in different modes of operation.

VI. Operating Conditions

FIGS. 5A-5C illustrate various operating conditions that a farmingmachine may experience, in accordance with an embodiment. Theillustrated farming machine 500 is substantially similar to the farmingmachine 100. The farming machine 500 includes an array of upward facinglight sensors 510 and an array of downward facing image sensors 520coupled to the mounting mechanism 140. In alternative embodiments, thefarming machine 500 can include a different number and/or configurationof light sensors 510 and/or image sensors 520. The light sensors 510 areconfigured to measure a characteristic of light incident on the sensors.As described above, the control system 130 adjusts one or more settingsof an image sensor 520 based on characteristics determined by a lightsensor in order to increase image quality and accurately identify andtreat plants. The control system may adjust the settings based ondifferent operational conditions (e.g., light conditions). Some examplesof different operational conditions are described below.

FIG. 5A illustrates a farming machine 500 operating on a sunny day. On asunny day, the brightness of a captured image may be high, which mayaffect image processing and plant identification. As such, the controlsystem 130 adjusts settings of the image sensors 520 to enhance plantidentification accuracy. For example, the light sensors 510 measure alight intensity of the light 570 a. The measured light intensity isassociated with images having incorrect white balance which, in turn,causes the control system 130 to misidentify plants. As such, thecontrol system 130 may increase the shutter speed of the image sensors520 to compensate for images captured in overly bright operationalconditions. In some cases, light 570 a measured by light sensors 510 ona sunny day may not cause the control system 130 to modify settings ofthe image sensors 520.

FIG. 5B illustrates the farming machine 500 operating on a cloudy day.On a cloudy day, an image captured by an image sensor 520 may seem dark,thereby causing the control system 130 to misidentify plants. Toillustrate, the image sensors 520 are configured for operation on asunny day and, thereby, images captured by the image sensors 520 are toodark. This discrepancy is because the light 570 a on a sunny day isdifferent than the light 570 b on a cloudy day. As such, the light 570 bon a cloudy day may have different measured characteristics (e.g., alower intensity, cooler color temperature) than the light 570 a on asunny day. The control system 130 adjusts the settings of the imagesensors 520 according to the measured characteristic of light 570 b. Forexample, the control system 130 decreases the shutter speed of the imagesensors 520 such that the captured images appear less dark.

In the embodiments of FIGS. 5A-5B, each of the light sensors 510 in thearray of sensors receive the same light (e.g., 570 a, 570 b). As such,the control system 130 determines an average value of the characteristicof light (e.g., 570 a, 570 b) using the array of light sensors 510. Thecontrol system 130 collectively adjusts settings of the image sensors520 based on the average value. In alternative embodiments, the lightmeasured by each light sensor 510 may be used to adjust the settings ofa corresponding image sensor 520 (e.g., image sensor 520 a and lightsensor 510 a are a corresponding pair).

However, in some cases, light sensors do not receive the same light. Inthese examples, the control system 130 compares characteristics of lightmeasured by each sensor in the array of light sensors 510 to eliminateoutliers and/or improve plant identification and treatment accuracy incase of uneven light conditions. Uneven lighting conditions may occurwhen one or more light sensors malfunction, are low on battery, arecovered in dust, pass through the shade, etc. In one example, thecontrol system 130 eliminates readings outside of a threshold rangebased on a median or average value. In other examples, the controlsystem 130 determines a standard distribution of the values of acharacteristic of light measured by each light sensor 510 in an array.The control system 130 can eliminate one or more light sensors 510 basedon the standard distribution. Responsive to identifying one or moremeasurements to be eliminated, the control system 130 may ignore themeasurement, remove the light sensor that recorded the measurement fromsubsequent operation, alert the system (e.g., send a message to anoperator, automatically generate an error warning, etc.), or take anyother appropriate remedial action. In a similar manner, light sensorsmay include one or more pixels/detectors as described above. In thiscase, each individual light sensor may be able to assist in determininguneven lighting conditions, etc. For example, if a shadow lies acrossthe sensor and half of the detectors are receiving full light, whilehalf are receiving partial light.

FIG. 5C illustrates the farming machine operating in uneven lightingconditions. In this particular example, the farming machine 500 isoperating on a sunny day, and one or more of the light sensors 510 ispassing through shade. In the example, the light sensors 510 measure acharacteristic of light 570 c in the sun (e.g., sun group 512) differentthan the same characteristic of light measured by the light sensors 510in the shade 575 (e.g., shade group 514). For example, the sun group 512measures an intensity approximately 110,000 lux and the shade group 514measures an intensity of approximately 20,000 lux. Averaging themeasurements of a characteristic of light from the array of lightsensors 510 can lead to capturing images that are unsuitable foraccurate identification, and subsequent treatment, of a plant.

The farming machine may employ a variety of methods to avoid inaccurateidentification of plants caused by uneven lighting conditions. In anexample, the control system 130 compares a characteristic measured byeach of the light sensors 510 to determine a difference in the receivedlight between each of the light sensors in the array of light sensors.In some embodiments, responsive to determining the difference is above ameasurement threshold, the control system 130 selects a subset of thearray of light sensors 510 and measures a characteristic of light basedon the subset of the array of light sensors. The control system 130 canselect a subset of the array of light sensors 510 based on a time of dayof operation, by comparing measurements to one or more recentmeasurements recorded by the light sensors 510, comparing measurementsto a standard value, or according to any other suitable guidelines.Continuing with the previous example, if the measurement threshold is20,000 lux, the difference exceeds the measurement threshold. Responsiveto the difference exceeding the measurement threshold, the controlsystem 130 selects the sun group 512 to determine the characteristic oflight (e.g., the control system 130 determines an intensity of 110,000lux). Additionally, the control system 130 adjusts image acquisitionsettings based on the average of the measured characteristic of thesubset of the array of light sensors. In alternative embodiments,another suitable method of evaluating a characteristic of light may beused.

VI. Example Identification and Treatment During Different OperatingConditions

FIG. 6 is a flow chart illustrating a method of identifying and treatinga plant using a farming machine with an upward facing light sensor and adownward facing image sensor, according to one or more embodiments. Thesteps of FIG. 6 are illustrated from the perspective of a system (e.g.,control system 130) with a hardware processor performing the method 600.However, some or all of the steps may be performed by other systems orcomponents. In addition, in some embodiments, the steps may be performedin parallel, in different orders, or asynchronously, and in otherembodiments, different steps altogether may be performed.

A farming machine including an imaging system for identifying plants anda treatment mechanism for treating identified plants. Traditionally, thefarming machine has limited hours of operation. For example, as the sunsets in the afternoon, the intensity of available light decreases andthe color temperature darkens. Images captured by an imaging system willappear darker compared to images captured by the imaging system at anearlier time of day. Due to the darker images, the farming machinecannot accurately identify the plants. However, here, the farmingmachine includes an imaging system with an upward facing light sensorand downward facing image sensor. The farming machine is configured todetermine the characteristics of light in the environment in real timeand adjust the settings of the imaging system accordingly. The real timeadjustment of the imaging system allows the farming machine to captureimages and identify plants at times that were, traditionally, untenablefor the farming machine.

For example, a farming machine may employ method 600 to properlyidentify and treat plants at sunset. The farming machine is movingthrough a field to identify plants (e.g., a crop) and treat theidentified plants by spraying them with a growth promoter. To identifyplants, a light sensor (e.g., light sensor 300) of the farming machinedetects 610 a characteristic of light incident on the light sensor. Thelight sensor is mounted in a substantially upward orientation away fromthe field as described above in relation to FIGS. 4A-4D. As an example,the farming machine captures an intensity of light and/or colortemperature of the light on the light sensor.

A control system (e.g., control system 130) of the farming machineadjusts a setting of an image sensor (e.g., image capture module 224) ofthe imaging system. The image sensor is mounted in a substantiallydownwards orientation towards the plants. The imaging system captures620 an image of a plant in the field using the setting(s) for the imagesensor determined based on the measured light characteristic. Forexample, the control system 130 may adjust a shutter speed of the imagesensor based on the measured intensity of light and/or colortemperature.

The control system identifies 630 the plant in the image. In an example,the control system employs a plant identification model to identify theplant in the image. The plant identification model identifies latentinformation in the image indicating that one or more of the pixelsrepresent a plant. For example, a plant identification model identifiespixels in the image representing a soybean plant that is suitable fortreatment with a growth promoter. Adjusting exposure parameters of theimage sensors allows the plant to be properly identified and treated indifferent lighting conditions (e.g., sunset). For example, without thecapabilities described herein, the plant identification model may not beable to identify the soybean plant because the image, and the plant inthe image, are too dark to be discernable. In some embodiments, insteadof adjusting a setting of an image sensor, the control system capturesan image of a plant using default or pre-set settings, and performs oneor more image processing operations on the image based on the measuredlight characteristic in order to identify the plant within the image. Inyet other embodiments, the identification of the plant within the imageis performed based on the measured light characteristic.

The control system generates control instructions for the treatmentmechanisms such that the treatment mechanism apply 640 a treatment tothe identified plant as the farming machine travels past the plant inthe field. For example, the control system generates machineinstructions for a spray mechanisms that sprays growth promoter on theidentified soybean plant as the farming machine travels past the plantin the field.

IX. Control System

FIG. 7 is a block diagram illustrating components of an example machinefor reading and executing instructions from a machine-readable medium.Specifically, FIG. 7 shows a diagrammatic representation of controlsystem 130 in the example form of a computer system 700. The computersystem 700 can be used to execute instructions 724 (e.g., program codeor software) for causing the machine to perform any one or more of themethodologies (or processes) described herein. In alternativeembodiments, the machine operates as a standalone device or a connected(e.g., networked) device that connects to other machines. In a networkeddeployment, the machine may operate in the capacity of a server machineor a client machine in a server-client network environment, or as a peermachine in a peer-to-peer (or distributed) network environment.

The machine may be a server computer, a client computer, a personalcomputer (PC), a tablet PC, a set-top box (STB), a smartphone, aninternet of things (IoT) appliance, a network router, switch or bridge,or any machine capable of executing instructions 724 (sequential orotherwise) that specify actions to be taken by that machine. Further,while only a single machine is illustrated, the term “machine” shallalso be taken to include any collection of machines that individually orjointly execute instructions 724 to perform any one or more of themethodologies discussed herein.

The example computer system 700 includes one or more processing units(generally processor 702). The processor 702 is, for example, a centralprocessing unit (CPU), a graphics processing unit (GPU), a digitalsignal processor (DSP), a controller, a state machine, one or moreapplication specific integrated circuits (ASICs), one or moreradio-frequency integrated circuits (RFICs), or any combination ofthese. The computer system 700 also includes a main memory 704. Thecomputer system may include a storage unit 716. The processor 702,memory 704, and the storage unit 716 communicate via a bus 708.

In addition, the computer system 700 can include a static memory 706, agraphics display 710 (e.g., to drive a plasma display panel (PDP), aliquid crystal display (LCD), or a projector). The computer system 700may also include alphanumeric input device 712 (e.g., a keyboard), acursor control device 714 (e.g., a mouse, a trackball, a joystick, amotion sensor, or other pointing instrument), a signal generation device718 (e.g., a speaker), and a network interface device 720, which alsoare configured to communicate via the bus 708.

The storage unit 716 includes a machine-readable medium 722 on which isstored instructions 724 (e.g., software) embodying any one or more ofthe methodologies or functions described herein. For example, theinstructions 724 may include the functionalities of modules of thesystem 130 described in FIG. 2 . The instructions 724 may also reside,completely or at least partially, within the main memory 704 or withinthe processor 702 (e.g., within a processor's cache memory) duringexecution thereof by the computer system 700, the main memory 704 andthe processor 702 also constituting machine-readable media. Theinstructions 724 may be transmitted or received over a network 726 viathe network interface device 720.

The control system 130 can comprise a processing unit (e.g., one or moreof a CPU, a GPU, or an FPGA) and a data storage medium (e.g., static ordynamic memory). In one embodiment, the control system 130 comprises adeep-learning GPU that is configured to effectively execute adeep-learning neural network. For example, the computer system 700 mayinclude an NVIDIA GeForce® GTX™ TITAN X using the Caffe deep learningframework or the NVIDIA Tx1 or Tx2 using the Tensorflow deep learningframework. Furthermore, image data passed in to the computerinstructions may be transmitted to the control system 130 for processingusing any type of transmission protocol. For example, the open systemsinterconnect (OSI) model may be used to send image data from the imageacquisition system 220 to the control system 130 using ethernetconnections between these components.

X. Additional Considerations

In the description above, for purposes of explanation, numerous specificdetails are set forth in order to provide a thorough understanding ofthe illustrated system and its operations. It will be apparent, however,to one skilled in the art that the system can be operated without thesespecific details. In other instances, structures and devices are shownin block diagram form in order to avoid obscuring the system.

Reference in the specification to “one embodiment” or “an embodiment”means that a particular feature, structure, or characteristic describedin connection with the embodiment is included in at least one embodimentof the system. The appearances of the phrase “in one embodiment” invarious places in the specification are not necessarily all referring tothe same embodiment.

Some portions of the detailed descriptions are presented in terms ofalgorithms or models and symbolic representations of operations on databits within a computer memory. An algorithm is here, and generally,conceived to be steps leading to a desired result. The steps are thoserequiring physical transformations or manipulations of physicalquantities. Usually, though not necessarily, these quantities take theform of electrical or magnetic signals capable of being stored,transferred, combined, compared, and otherwise manipulated. It hasproven convenient at times, principally for reasons of common usage, torefer to these signals as bits, values, elements, symbols, characters,terms, numbers, or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the following discussion,it is appreciated that throughout the description, discussions utilizingterms such as “processing” or “computing” or “calculating” or“determining” or “displaying” or the like, refer to the action andprocesses of a computer system, or similar electronic computing device,that manipulates and transforms data represented as physical(electronic) quantities within the computer system's registers andmemories into other data similarly represented as physical quantitieswithin the computer system memories or registers or other suchinformation storage, transmission or display devices.

Some of the operations described herein are performed by a computerphysically mounted within a farming machine 100. This computer may bespecially constructed for the required purposes, or it may comprise ageneral-purpose computer selectively activated or reconfigured by acomputer program stored in the computer. Such a computer program may bestored in a computer readable storage medium, such as, but is notlimited to, any type of disk including floppy disks, optical disks,CD-ROMs, and magnetic-optical disks, read-only memories (ROMs), randomaccess memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, orany type of non-transitory computer readable storage medium suitable forstoring electronic instructions.

The figures and the description above relate to various embodiments byway of illustration only. It should be noted that from the followingdiscussion, alternative embodiments of the structures and methodsdisclosed herein will be readily recognized as viable alternatives thatmay be employed without departing from the principles of what isclaimed.

One or more embodiments have been described above, examples of which areillustrated in the accompanying figures. It is noted that whereverpracticable similar or like reference numbers may be used in the figuresand may indicate similar or like functionality. The figures depictembodiments of the disclosed system (or method) for purposes ofillustration only. One skilled in the art will readily recognize fromthe following description that alternative embodiments of the structuresand methods illustrated herein may be employed without departing fromthe principles described herein.

Some embodiments may be described using the expression “coupled” and“connected” along with their derivatives. It should be understood thatthese terms are not intended as synonyms for each other. For example,some embodiments may be described using the term “connected” to indicatethat two or more elements are in direct physical or electrical contactwith each other. In another example, some embodiments may be describedusing the term “coupled” to indicate that two or more elements are indirect physical or electrical contact. The term “coupled,” however, mayalso mean that two or more elements are not in direct physical orelectrical contact with each other, but yet still co-operate or interactwith each other. The embodiments are not limited in this context.

As used herein, the terms “comprises,” “comprising,” “includes,”“including,” “has,” “having” or any other variation thereof, areintended to cover a non-exclusive inclusion. For example, a process,method, article or apparatus that comprises a list of elements is notnecessarily limited to only those elements but may include otherelements not expressly listed or inherent to such process, method,article or apparatus. Further, unless expressly stated to the contrary,“or” refers to an inclusive or and not to an exclusive or. For example,a condition A or B is satisfied by any one of the following: A is true(or present) and B is false (or not present), A is false (or notpresent) and B is true (or present), and both A and B is true (orpresent).

In addition, use of the “a” or “an” are employed to describe elementsand components of the embodiments herein. This is done merely forconvenience and to give a general sense of the system. This descriptionshould be read to include one or at least one and the singular alsoincludes the plural unless it is obvious that it is meant otherwise.

Upon reading this disclosure, those of skill in the art will appreciatestill additional alternative structural and functional designs for asystem and a process for identifying and treating plants with a farmingmachine including a control system executing a semantic segmentationmodel. Thus, while particular embodiments and applications have beenillustrated and described, it is to be understood that the disclosedembodiments are not limited to the precise construction and componentsdisclosed herein. Various modifications, changes and variations, whichwill be apparent to those, skilled in the art, may be made in thearrangement, operation and details of the method and apparatus disclosedherein without departing from the spirit and scope defined in theappended claims.

The invention claimed is:
 1. A farming machine comprising: an imageacquisition system configured to capture an image of a plant in a fieldas the farming machine moves through the field, the image acquisitionsystem capturing the image using an image sensor directed in a downwardsorientation towards the plant; a light measurement system configured tomeasure light incident on the light measurement system; a plurality ofplant treatment mechanisms for treating the plant in the field as thefarming machine moves past the plant in the field; and a control systemincluding a processor, the processor configured to: capture the image ofthe plant in the field using the image acquisition system and based onlight measured by the light measurement system by: determining an actualreflectance of the plant, determining a perceived reflectance of theplant, and adjusting a configuration of the image acquisition system bymodifying one or more exposure parameters of the image acquisitionsystem based on a difference between the actual reflectance and theperceived reflectance of the plant, identify the plant in the imagebased on pixels in the image identified as representing the plant andimage properties of the captured image corresponding to theconfiguration of the image acquisition system, and actuate one or moreof the plurality of plant treatment mechanisms to treat the identifiedplant.
 2. The farming machine of claim 1, wherein the image acquisitionsystem further comprises: one or more additional image sensors, each ofthe additional image sensors directed in the downwards orientation. 3.The farming machine of claim 1, wherein the light measurement systemcomprises one or more light sensors.
 4. The farming machine of claim 1,wherein: a light sensor is coupled to a top surface of the farmingmachine, and the image sensor is coupled to a bottom surface of thefarming machine.
 5. The farming machine of claim 1, further comprising:a mounting mechanism having a first length that is substantiallyorthogonal to a direction of movement of the farming machine, andwherein the plurality of treatment mechanisms are coupled to themounting mechanism.
 6. The farming machine of claim 5, wherein: theimage acquisition system is mounted to the mounting mechanism, and thelight measurement system is mounted to the mounting mechanism.
 7. Thefarming machine of claim 1, wherein the light measurement systemcomprises: an electronic subsystem coupled to a light sensor; a baseincluding a receptacle configured to retain the electronic subsystem andthe light sensor, the receptacle recessed within at least one of a setof surfaces of the base; and a cap coupled to the base wherein the capencloses the electronic subsystem and the light sensor.
 8. The farmingmachine of claim 7, wherein the image sensor is one of: coupled to afront surface of the farming machine and directed downwards, coupled toa bottom surface of the farming machine and directed downwards, coupledto the bottom surface of the farming machine and titled forward, coupledto a top surface of the farming machine and direct downwards, andcoupled to a side of the farming machine and directed downwards.
 9. Afarming machine comprising: a light measurement system configured tomeasure a characteristic of light; an image acquisition systemconfigured to capture an image of a plant in a field; a plurality ofplant treatment mechanisms for treating plants in the field; and acontrol system for: capturing an image of the plant in the field usingthe image acquisition system by: determining an actual reflectance ofthe plant, determining a perceived reflectance of the plant at least inpart using the light measurement system, and adjusting the configurationof the image acquisition system based on a difference between the actualreflectance and the perceived reflectance of the plant; and identifyingthe plant in the image and actuating one or more of the plurality ofplant treatment mechanisms to treat the identified plant based on pixelsin the image identified as representing the plant.
 10. The farmingmachine of claim 9, wherein the image acquisition system comprises oneor more image sensors, each of the image sensors facing in a downwardsdirection towards the plant as the farming machine moves through thefield.
 11. The farming machine of claim 9, wherein the light measurementsystem comprises one or more light sensors.
 12. A method comprising:detecting an amount of light incident on a light sensor mounted on afarming machine passing through a field; capturing an image of a plantin the field using an image sensor mounted on the farming machine, theimage captured using one or more exposure parameters for the imagesensor selected based, by: determining an actual reflectance of theplant, determining a perceived reflectance of the plant, and adjustingthe one or more exposure parameters based on a difference between theactual reflectance and the perceived reflectance of the plant;identifying the plant in the image based on pixels in the imageidentified as representing the plant; and applying, with a treatmentmechanism mounted on the farming machine, a treatment to the identifiedplant.
 13. The method of claim 12, wherein the image sensor isorientated to face downward towards the plant in the field.
 14. Themethod of claim 12, wherein adjusting the one or more exposureparameters includes: determining the difference exceeds a differencethreshold; and responsive to the difference exceeding the differencethreshold, adjusting an exposure parameter of the one or more exposureparameters.
 15. The method of claim 12, wherein the one or more exposureparameters is one of: shutter speed, aperture, ISO speed, or whitebalance.
 16. The method of claim 12, further comprising: evaluating acharacteristic of the detected light; determining whether thecharacteristic of the detected light exceeds a threshold correspondingto the characteristic; and responsive to determining the characteristicexceeds the threshold corresponding to the characteristic, adjusting theone or more exposure parameters.
 17. The method of claim 16, wherein thecharacteristic is one of: light intensity or color temperature.
 18. Themethod of claim 12, further comprising: determining a characteristic oflight incident on a plurality of light sensors mounted on the farmingmachine; and comparing the characteristic of light incident on each ofthe plurality of light sensors to determine a difference in thecharacteristic of light incident on each of the plurality of sensors.19. The method of claim 18, further comprising: selecting a subset ofthe plurality of light sensors based on the characteristic of lightincident on each of the plurality of light sensors and the difference inthe characteristic of light incident on each of the plurality ofsensors; and evaluating the characteristic of light based on the subsetof the plurality of light sensors.
 20. The method of claim 19, furthercomprising adjusting the one or more exposure parameters based on thecharacteristic of light evaluated by the subset of the plurality oflight sensors.